The present application relates to image processing, and more specifically, to reconstruction of scene and camera parameters from image data. Estimating the accurate positions of cameras and locations of three-dimensional (3D) scene points from a collection of images obtained by cameras is a classic problem in computer vision, referred to as structure from motion (SfM). The problem of optimizing for the camera parameters and scene points using the corresponding points in images is an important component of SfM and is also referred to as Bundle Adjustment (BA).